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Abstract. Image segmentation techniques are predominately based on 
parameter-laden optimization. The objective function typically involves 
weights for balancing competing image fidelity and segmentation regular- 
ization cost terms. Setting these weights suitably has been a painstaking, 
empirical process. Even if such ideal weights are found for a novel image, 
most current approaches fix the weight across the whole image domain, 
ignoring the spatially- varying properties of object shape and image ap- 
pearance. We propose a novel technique that autonomously balances 
these terms in a spatially-adaptive manner through the incorporation of 
image reliability in a graph-based segmentation framework. We validate 
on synthetic data achieving a reduction in mean error of 47% (p-value 
<< 0.05) when compared to the best fixed parameter segmentation. We 
also present results on medical images (including segmentations of the 
corpus callosum and brain tissue in MRI data) and on natural images. 

Key words: Adaptive regularization, adaptive weights, image segmen- 
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1 Introduction 

Robust automated image segmentation is a highly desirable goal that contin- 
ues to defy solution. In medical images for example, natural and pathological 
variability may result in complicated and unpredictable image and shape fea- 
tures. Current segmentation methods are predominantly based on optimization 
procedures that produce so called 'optimal' segmentations at their minimum. 
Optimization methods typically incorporate a tradeoff between two classes of 
cost terms: data fidelity and regularization. This actually is the case not only in 
segmentation, but also in image registration, shape matching, and other com- 
puter vision tasks. This basic tradeoff scheme is ubiquitous, relating to Occam's 
razor and Akaike/Bayesian information criteria [I], and is seen in many forms, 
such as likelihood versus prior in Bayesian methods and loss versus penalty 
in machine learning [3]. Therefore, any advancement in controlling the balance 



2 J. Rao, G. Hamarneh, and R. Abugharbieh 



between competing cost terms will benefit many related applications and algo- 
rithmic formulations in medical image analysis. Optimization-based segmenta- 
tion methods that are fragile and highly sensitive to this tradeoff are plentiful, 
including active contours techniques [4 ^ [5] [7] , graph cut methods [5] , and opti- 
mal path approaches [H]- For simplifying the exposition of the ideas in this paper, 
we will adopt the simplified but general form of the cost or energy function: 

E{S\I, a, p) = aE,nt{S) + PE,,t{.S\I) (1) 

where S is the segmentation and / is the image. Eint is the internal cost term 
contributing to the regularization of the segmentation, most often by enforcing 
some smoothness constraints, in order to counteract the effects of imaging ar- 
tifacts. Ef,a:t is the external cost term contributing to the contour's conformity 
to desired image features, e.g., edges. The weights a and fi are typically set 
empirically by the users based on their judgment of how to best balance the 
requirements for regularization and adherence to image content. In most cases, 
this is a very difficult task and the parameters may be unintuitive for a typical 
non-technical end user, e.g. a clinician, who lacks knowledge of the underlying 
algorithm's inner working. Also the resultant segmentations can vary drastically 
based on how this balance is set. Avoiding the practice of ad-hoc setting of such 
weights is the driving motivation for our work here. 

To the best of our knowledge, regularization weights have traditionally been 
determined empirically and are fixed across the image domain (i.e. do not vary 
spatially). In Pluempitiwiriyawej et al [7,, the weights are changed as the opti- 
mization progresses, albeit in an ad-hoc predetermined manner. Mcintosh and 
Hamarneh [TU] demonstrated that adapting the regularization weights across 
a set of images is necessary in addressing the variability in real clinical image 
data. However, neither approach varies the weights spatially across the image 
and hence are not responsive or adaptive to local features within a single image. 

Image regions with noise, weak or missing boundaries, and/or occlusions are 
commonly encountered in real image data. For example, degradation in medi- 
cal images can occur due to tissue heterogeneity ("graded decomposition" [IT]), 
patient motion, or imaging artifacts, e.g. echo dropouts in ultrasound or non- 
uniformity in magnetic resonance. In such cases and in order to increase seg- 
mentation robustness and accuracy, more regularization is needed in less reliable 
image regions which suffer from greater deterioration. Although an optimal reg- 
ularization weight can be found for a single image in a set [TP] , the same weight 
may not be optimal for all regions of that image. Spatially adapting the regular- 
ization weights provide greater control over the segmentation result, allowing it 
to adapt not only to images with spatially varying noise levels and edge strength, 
but also to objects with spatially-varying shape characteristics, e.g. smooth in 
some parts and jagged in others. 

Some form of spatially adaptive regularization over a single image appeared 
in a recent work by Dong et al [T2]. For segmenting an aneurysm, they varied 
the amount of regularization based on the surface curvature of a pre-segmented 
vessel. The results demonstrated improvements due to adaptive regularization. 
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However, the regularization weights did not rely on the properties of the image 
itself, which limited the generality of the method. Kokkinos et al [13] investigated 
the use of adaptive weights for the task of separating edge areas from textured 
regions using a probabilistic framework, where the posterior probabilities of edge, 
texture, and smoothness cues were used as weights for curve evolution. Similarily, 
Malik et al [H] and, very recently, Erdem and Tari [15 tackled the problem of 
texture separation and selected weights based on data cues. However, while these 
methods focused on curve evolution frameworks, our current work focuses on 
graph-based segmentation. Additionally, we emphasize balancing the cost terms 
by adapting regularization for images plagued by noisy and weak or diffused 
edge problems rather than textured patterns in natural images, which we leave 
as future work. 

In this paper, we advocate the strong need for spatially-adaptive balancing of 
cost terms in an automated, robust, data-driven manner to relax the requirement 
on the user to painstakingly tweak these parameters. We also demonstrate how 
existing fixed- weight approaches (even if globally optimized) are often inadequate 
for achieving accurate segmentation. To address the problem, we propose a novel 
data-driven method for spatial adaptation of optimization weights. We develop 
a new spectral flatness measure of local image noise to balance the energy cost 
terms at every pixel, without any prior knowledge or fine-tuning. 

We validate our method on synthetic, medical, and natural images and com- 
pare its performance against two alternative approaches for regularization: us- 
ing the best possible spatially- fixed weight, and using the globally optimal set of 
spatially-varying weights as found automatically through dynamic programming. 

The rest of this paper is organized as follows; Section [2] presents a brief 
overview of our segmentation process, the formulation for our proposed relia- 
bility method along with a formulation of the globally optimum graph search 
approach. Section [3] presents qualitative comparisons of our method to both 
globally-optimal and fixed parameter-based methods and reports quantitative 
analysis of the resulting error. Section [4] presents our conclusions and an overview 
of future work planned. 

2 Methods 

Our formulation employs energy-minimizing boundary-based segmentation, where 
the objective is to find a contour that correctly separates an object from back- 
ground. We begin by formulating the energy of a contour and specifying how the 
regularization term is weighted in our definition. We then present our approach 
for a data-driven spatially adaptive regularization method, and end the section 
with a brief discussion of the globally-optimum parameter method. 

2.1 Energy-Minimizing Segmentation 

We embed a parametric contour C{q) = C{x{q),y{q)) : [0,1] ^ i7 C in 
image / : 12 ^ R. We use a single adaptive weight w{q) G [0, 1] that varies over 
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the length of the contour and re- write ([T]) as: 

E{C{q),w{q)) - {w{q)E,^t{C{q)) + (1 - w{q))E,,t{C{q))) dq (2) 
Jo 

where 

E,,t{C{q)) = 1 - |V/(C(g))|/max|V/(C(g))| (3) 

penalizes weak boundaries and 

E^nt{C{q)) = \dC{q)/dq\ (4) 

is the length of the contour. To minimize E with respect to C{q) in (|2]), we adopt 
a discrete formulation of the optimization problem. We model the image as a 
graph where each pixel is represented by a vertex Vi and edges Cij = {vi, Vj) that 
capture the pixel's connectedness (e.g. 8-connectedness in 2D images). A local 
cost 

Cij = C{eij) = wEint {vi,Vj) + (1 - W) Eext{Vi) (5) 

is assigned to each edge Cij, where Eint {vi, Vj) is the Euclidean distances between 
Vi and Vj (e.g. 1 for 4-connected neighbors and %/2 for diagonal neighbors in 2D). 
The contour that minimizes the total energy E ^ ^ Cij represents the optimal 

solution for the segmentation and is found by solving a minimal path problem, 
similar to [5], using Dijkstra's algorithm [16j . 



2.2 Automatic Spatially- Adaptive Balancing of Energy Cost Terms 

Our approach for balancing the cost terms is to gauge the levels of signal vs. 
noise in local image regions. We estimate the edge evidence G{x, y) and noise 
level N{x, y) in each region of the image and set w{x, y) in ^ such that regions 
with high noise and low boundary evidence (i.e. low reliability) have greater 
regularization, and vice versa. Hence, w(x,y) is mapped to image reliability 
R{x, y) as 

w{x,y) = l-R{x,y) (6) 

where 

R{x,y)^{l-N{x,y))G{x,y). (7) 

Assuming additive white noise, uncorrelated between pixels, we estimate spatially- 
varying noise levels N{x,y) using local image spectral flatness (SF). SF is a 
well-known Fourier-domain measure that has been employed in audio signal 
processing and compression applications |17j |18j . SF exploits the property that 
white noise exhibits similar power levels in all spectral bands and results in a 
flat power spectrum, whereas uncorrupted signals have power concentrated in 
certain spectral bands and result in a more impulse-like power spectrum. We 
extend the SF measure to 2D and measure N(x^ y) as 

exp ( 4^ J,!"^ In S {ujx , ) duj^dujy ) 
y) = \ rn ^, r-r—, (8) 

4^ 5* {UJ^,UJy) duj^dujy 
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where S{LL!x,i-Oy) is the 2D power spectrum of the image and (aJxTi^y) are spatial 
frequencies. We use G{x,y) = ma,x{\\/Ix{x,y)\ ,\\/Iy{x,y)\), where \/Ix{x,y) 
and \JIy{x,y) represent the x and y components of the image gradient. We 
chose this measure rather than the standard gradient magnitude for its rotational 
invariance in the discrete domain. 

2.3 Non-Contextual Globally Optimal Weights 

A theoretically appealing and intuitive approach for setting the regularization 
weight is to optimize i? in (|2]) for the weight w{q) itself in addition to opti- 
mizing the contour. In our discrete setting, this involves a 'three dimensional' 
graph search that computes the globally optimal, spatially-adaptive regulariza- 
tion weight wi^q), in conjunction with the contour's spatial coordinates, i.e. we 
optimiz(|jC'((7) = {x{q),y{q),w{q)). 

In this formulation, each vertex in the original graph is now replaced by K 
vertices representing the different choices of the weight value at each pixel. In 
addition, graph edges now connect vertices corresponding to neighboring image 
pixels for all possible weights. Note that the optimal path C{q) cannot pass 
through the same {x{q),y{q)) for different w, i.e. only a single weight can be 
assigned per pixel. Our graph search abides by this simple and logical constraint. 
The optimal C{q) and 'w{q) that globally minimize ([2| are again calculated using 
dynamic programming but now on this new (x,y,w) graph. 

There are, however, three main drawbacks to this globally optimum (in 
{x, y, w)) method: (i) it does not explicitly encode image reliability, even though 
regularization is essential in regions with low reliability; (ii) this approach will 
encourage a bimodal behavior of the regularization weight: 

= otherwise /' 

and (iii) combining the weight and segmentation optimization into one process 
reduces the generality of the method. In short, even though optimal with re- 
spect to ii^ in the solution is incorrect and, as we later demonstrate, inferior 
to the spatially adaptive balancing of energy cost terms proposed in Sec. |2.1[ 
Furthermore, by combining the weight optimization and contour optimization 
processes into one, we reduce the generality of the method. Finding globally- 
optimal weights for other segmentation frameworks would require significant 
changes to the energy minimization process. 

3 Results and Discussion 

We first performed quantitative tests on 16 synthetic images carefully designed 
to cover extreme shape and appearance variations (one example is shown in Fig. 

^ This is similar in spirit to 19 and 20:_ where they also optimize for a non-spatial 
variable: vessel radius or scale, in addition to the spatial coordinates of the segmen- 
tation contour. 
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[TJ. The test data was created by modeling an object boundary as a sinusoidal 
function with spatially-varying frequency to simulate varying contour smooth- 
ness conditions. We then added spatially-varying (non-stationary) additive white 
Gaussian noise patterns of increasing variance. We also spatially varied the gradi- 
ent magnitude of the object boundary across each image by applying Gaussian 
blurring kernels at different scales in different locations. Our resulting image 
reliability measure is exemplified in Fig. [l] for two synthetic images with the re- 
sulting segmentations shown in Fig. |2] We also show the contour obtained using 
the globally-optimal weights method (Sec. |2.3| , and the contour obtained using 
a spatially -fixed regularization weight, set to the value producing the smallest 
(via brute force search) segmentation error. 



(a) Synthetic sinusoidal image I{x, y) 


(b) Synthetic sinusoidal image I{x, y) 






(c) Edge evidence measure G{x, y) 


(d) Edge evidence measure G{x, y) 






(e) Noise level estimate N{x, y) 


(f ) Noise level estimate N{x, y) 








(g) Total reliability measure R(x, y) 


(h) Total reliability measure R{x^ y) 



Fig. 1. Synthetic image with spatially varying noise and blurring (both increas- 
ing from right to left) and with changing boundary smoothness (smooth on the 
left and jagged on the right). Black intensities corresponds to and white to 1. 
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We quantitatively examined our method's performance using ANOVA test- 
ing on 25 noise realizations of each image in the dataset, where the error was 
determined by the Hausdorff distance to the ground truth contour. Our method 
resulted in a mean error (in pixels) of 6.33 (std. dev. 1.36), whereas the best 
fixed-weight method had a mean error of 12.05 (std. dev. 1.61), and the globally- 
optimum weight method had a mean error of 33.06 (std. dev. 3.66) . Furthermore, 
for each image, we found our method to be significantly more accurate with all 
p values << 0.05. 




Fig. 2. Color is essential for proper viewing, please refer to the e-copy. Con- 
tours obtained from: (red) proposed adaptive weights, (green) lowest-error fixed 
weight, and (cyan) globally optimum weight. 



We also tested our method on clinical MR images of the corpus callosum 
(CC) , which exhibits the known problem of a weak boundary where the CC meets 
the fornix (Fig. |3(a) I. Note how the contour obtained using globally optimal 
weights exhibits an optimal, yet undesirable, bimodal behavior (either blue or red 
in Fig. 3(a) I completely favoring only one of the terms at a time. In comparison, 
our method automatically boosts up the regularization (stronger red in Fig. |3(b)[ ) 
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at the CC-fornix boundary producing a better delineation. The segmentation 
results of all three methods for the same image are shown in Fig. |4] 




(a) (b) 

Fig. 3. (Color figure, refer to e-copy). Results of (a) globally-optimum weight 
method and (b) proposed adaptive-weight method for a corpus callosum MR 
image. The coloring of the contours reflects the value of the spatially-adaptive 
weight. The same color map is used for both flgures, with pure blue corresponding 
to w ~ for pure red for and w = 1. 

We also tested our method on MR data from Brain Web "IT]. Fig. [5] shows the 
segmentation of the cortical surface in a proton density (PD) image with a noise 
level of 5%. This example is a difficult scenario due to the high level of noise and 
low resolution of the image. Our proposed method provided a smoother contour 
while conforming to the cortical boundary when compared to the other methods 
(although the difference was not too large). 

We also tested our method on natural images, such as the leaf shown in Fig. 
6(a)[ The resulting reliability measure (Fig. |6(b)| has lower image reliability and, 
hence, higher regularization at the regions of the leaf obscured by snow, whereas 
reliable boundaries light up (bright white boundary segments). The resulting 
segmentations are shown in Fig. [7] 

4 Conclusion 

We proposed a novel approach for addressing a ubiquitous problem that plagues 
many energy minimization segmentation techniques; how to balance the weights 
of competing energy terms. Our technique spatially adapts the regularization 
weight based on local spectral-flatness and data-driven image reliability mea- 
sures. We emphasize two contributions of our work: (i) regularization must vary 
spatially and must increase where image evidence is less reliable, and (ii) we 
discriminate between signal edges (object boundaries) versus noise edges by 
extending a spectral flatness measure that is well established in audio signal 
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Fig. 4. {Color figure, refer to e-copy). Segmentation results of the CC in the MR 
image of Fig.|3] Green contour is result of the proposed spatially adaptive weight 
method, red contour is result of best fixed-weight method, and cyan contour is 
result of the globally-optimum weight method. 




Fig. 5. (Color figure, refer to e-copy). Contours produced by using proposed 
adaptive weight (red), globally-optimum weight (cyan), best fixed-weight (blue). 
Ground truth contour is shown in green. Note the improved regularization using 
our method. 
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Fig. 7. (Color figure, refer to e-copy). Segmentation results of the leaf in Fig. 
6 (a) I from our method (green), fixed-weight of 1 (black), 0.5 (blue), and (red). 
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processing to 2D. By analyzing the spectral flatness of the observed signal, a 
spatially-varying evidence of signal versus noise is derived fully automatically 
(without any tuning) , which we use to spatially adapt the regularization using a 
clear reliability metric. We note that there is no restriction against using other 
noise estimation methods in our proposed reliability-modulated regularization, 
making our method of broader interest. Using a large synthetic dataset exhibiting 
extreme variations of image deterioration and boundary shape, we demonstrated 
statistically significant reduction in segmentation error compared to using the 
best fixed weight, and to a globally-optimal, spatially- varying approach that uses 
dynamic programming to optimize for the regularization weight in conjunction 
with contour. 

Our current work focused on minimal-path approaches for segmentation. 
However, our approach of reliability-based regularization is applicable to a wide 
range of energy-minimizing segmentation techniques. We are currently extend- 
ing our approach to other variational and graph-based segmentation approaches 
such as [3] [H^ [22^ . Additionally, we intend to expand our technique to handle en- 
ergy functionals where multiple weights balance the different energy terms and 
to explore alternative reliability measures. Another important conclusion from 
our work is that globally optimal weights do not appear to be desirable. We 
intend to further explore this issue in more detail in the future. 
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